Vehicle-mounted image target objection recognition device

ABSTRACT

A vehicle-mounted image processing device  1000  acquires an image around a host vehicle  10 , senses a feature amount of a target object present on a road surface from the image, and relative position coordinates with respect to the host vehicle  10 , acquires information of a three-dimensional object around the host vehicle  10  and relative position coordinates of the three-dimensional object, distinguishes whether the feature amount is a feature amount of a feature on the road surface or a feature amount of a feature on the three-dimensional object by using a positional relationship between the feature amount and the three-dimensional object, and recognizes the target object by using a feature on the road surface. As a result, a vehicle-mounted image processing device, which is capable of avoiding misrecognition of a part of a three-dimensional object as a target object when sensing the target object present on a road surface such as a parking bay by using a camera image, is provided.

TECHNICAL FIELD

The present invention relates to a vehicle-mounted image processingdevice which is suitably used when a vehicle control for assisting in adriving operation of a driver is performed, or the like.

BACKGROUND ART

PTL 1 discloses a parking bay sensing device including: imaging meansthat captures an image of an area behind a vehicle; edge extractionmeans that extracts edges from the captured image; image conversionmeans that converts the image with the edges extracted therefrom into anoverhead image; area division means that divides the converted overheadimage into left and right areas; straight line sensing means that sensesstraight lines in each area by Hough transform from the divided left andright areas; first determination means that determines whether or notthe sensed straight line is an end of a line having a width; and seconddetermination means that creates a combination of two straight lines foreach of the left and right areas from a plurality of the straight linesdetermined to be an end of a line having a width, and determines whetheror not the combination corresponds to both ends of a line having awidth, for the purpose of accurately sensing straight lines to becandidates of both left and right ends of a line of a parking bay,preventing erroneous sensing of an unrelated straight line as both endsof the line of the parking bay, and accurately sensing the parking bay.

CITATION LIST Patent Literature

PTL 1: JP 2012-80497 A

SUMMARY OF INVENTION Technical Problem

In recent years, a system of sensing a parking bay by using a camera tosupport a parking operation of a driver has been developed. For example,an autonomous parking system of sensing a parking bay around a hostvehicle and automatically performing a parking operation of a driverpartially or totally, and the like have been commercialized.

In a case where a parking bay is sensed by using a camera, the sensingis performed by using edge information generated based on a differencein a brightness between a white line and a road surface. In this case,when there is a shadow near the white line of the parking bay, an edgegenerated due to the shadow is misrecognized as the parking bay in somecases. As a countermeasure for such a misrecognition, for example, PTL 1described above discloses a technology of checking a falling edge whichis present near a rising edge by using a fact that a white line has twoedges with different brightness change directions, thereby preventing aninfluence of an edge generated by a shadow.

However, three-dimensional object information cannot be obtained from acamera image. For this reason, for example, a bumper, a side sill, acoating, or the like of a parked car which is adjacent to a parking baylooks the same as a white line of the parking bay in the image and arising edge and a falling edge are also paired, such that an edge of theside sill and the white line are paired to thereby be misrecognized asthe parking bay. That is, a part of the three-dimensional object ismisrecognized as the parking bay, which is problematic.

The present invention has been made in view of the problems describedabove, and an object of the present invention is to provide avehicle-mounted image processing device capable of avoidingmisrecognition of a part of a three-dimensional object as a targetobject when sensing the target object present on a road surface such asa parking bay by using a camera image.

Solution to Problem

In order to solve the above problems, for example, configurationsdescribed in the claims are adopted.

The present invention includes a plurality of means for solving theproblems described above, and an example of the means is avehicle-mounted image processing device recognizing a target objectaround a host vehicle, the vehicle-mounted image processing deviceincluding: an image acquisition unit which acquires an image around thehost vehicle captured by an imaging unit; a feature amount extractionunit which extracts, from the image around the host vehicle acquired bythe image acquisition unit, a feature amount of the target object andcoordinate information of the feature amount with respect to the hostvehicle when it is assumed that a feature having the feature amount ison a road surface; a three-dimensional object information storage unitwhich acquires and stores coordinate information of a three-dimensionalobject around the host vehicle with respect to the host vehicle; afeature amount distinguishing unit which distinguishes whether thefeature amount is a feature amount of a feature on the road surface or afeature amount of a feature on the three-dimensional object by using apositional relationship between the coordinate information of thefeature amount sensed by the feature amount extraction unit with respectto the host vehicle and the coordinate information of thethree-dimensional object stored in the three-dimensional objectinformation storage unit with respect to the host vehicle; and a targetobject recognition unit which recognizes the target object by using thefeature amount distinguished as a feature amount of a feature on theroad surface by the feature amount distinguishing unit.

Advantageous Effects of Invention

According to the present invention, it is possible to avoidmisrecognition of a part of a three-dimensional object as a targetobject when sensing the target object present on a road surface such asa parking bay by using a camera image. Problems to be solved,configurations, and effects other than those described above areclarified from the description of the following embodiments.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a vehicle-mounted image processing deviceaccording to a first embodiment of the present invention.

FIG. 2 is a view illustrating an example of a processing performed by animage acquisition unit according to the first embodiment.

FIG. 3 is a flowchart illustrating a processing performed by a whiteline feature sensing unit according to the first embodiment.

FIG. 4 is a flowchart illustrating a processing performed by athree-dimensional object sensing unit according to the first embodiment.

FIG. 5A is a diagram for describing a processing performed by thethree-dimensional object sensing unit according to the first embodiment.

FIG. 5B is a diagram for describing a processing performed by thethree-dimensional object sensing unit according to the first embodiment.

FIG. 6 is a flowchart illustrating a processing performed by athree-dimensional object information storage unit according to the firstembodiment.

FIG. 7A is a diagram for describing a processing performed by thethree-dimensional object information storage unit according to the firstembodiment.

FIG. 7B is a diagram for describing a processing performed by thethree-dimensional object information storage unit according to the firstembodiment.

FIG. 7C is a diagram for describing a processing performed by thethree-dimensional object information storage unit according to the firstembodiment.

FIG. 7D is a diagram for describing a processing performed by thethree-dimensional object information storage unit according to the firstembodiment.

FIG. 8 is a flowchart illustrating a processing performed by a whiteline feature distinguishing unit according to the first embodiment.

FIG. 9A is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the first embodiment.

FIG. 9B is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the first embodiment.

FIG. 9C is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the first embodiment.

FIG. 9D is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the first embodiment.

FIG. 9E is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the first embodiment.

FIG. 9F is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the first embodiment.

FIG. 10 is a flowchart illustrating a processing performed by a parkingbay recognition unit according to the first embodiment.

FIG. 11A is a diagram for describing a processing performed by theparking bay recognition unit according to the first embodiment.

FIG. 11B is a diagram for describing a processing performed by theparking bay recognition unit according to the first embodiment.

FIG. 12 is a block diagram of a first modified example of the firstembodiment.

FIG. 13 is a block diagram of a second modified example of the firstembodiment.

FIG. 14 is a block diagram of a third modified example of the firstembodiment.

FIG. 15 is a block diagram of a vehicle-mounted image processing deviceaccording to a second embodiment of the present invention.

FIG. 16 is a flowchart illustrating a processing performed by a whiteline feature distinguishing unit according to the second embodiment.

FIG. 17A is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the second embodiment.

FIG. 17B is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the second embodiment.

FIG. 17C is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the second embodiment.

FIG. 17D is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the second embodiment.

FIG. 17E is a diagram for describing a processing performed by the whiteline feature distinguishing unit according to the second embodiment.

FIG. 18 is a block diagram of a vehicle-mounted image processing deviceaccording to a third embodiment of the present invention.

FIG. 19 is a flowchart illustrating a processing performed by an endpoint feature sensing unit according to the third embodiment.

FIG. 20A is a diagram for describing a processing performed by the endpoint feature sensing unit according to the third embodiment.

FIG. 20B is a diagram for describing a processing performed by the endpoint feature sensing unit according to the third embodiment.

FIG. 21A is a diagram for describing a processing performed by an endpoint feature distinguishing unit according to the third embodiment.

FIG. 21B is a diagram for describing a processing performed by the endpoint feature distinguishing unit according to the third embodiment.

FIG. 21C is a diagram for describing a processing performed by the endpoint feature distinguishing unit according to the third embodiment.

FIG. 21D is a diagram for describing a processing performed by the endpoint feature distinguishing unit according to the third embodiment.

FIG. 22 is a flowchart illustrating a processing performed by a parkingbay recognition unit according to the third embodiment.

FIG. 23 is a flowchart illustrating an example of another processingperformed by the parking bay recognition unit according to the thirdembodiment.

FIG. 24 is a block diagram of a vehicle-mounted image processing deviceaccording to a fourth embodiment of the present invention.

FIG. 25 is a flowchart illustrating a processing performed by a roadsurface painting recognition unit according to the fourth embodiment.

FIG. 26 is a block diagram of a vehicle-mounted image processing deviceaccording to a fifth embodiment of the present invention.

FIG. 27 is a flowchart illustrating a processing performed by acurbstone recognition unit according to the fifth embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, a vehicle-mounted image processing device according to anembodiment of the present invention which senses a target object on theground such as a parking bay, a road surface painting, or a curbstonebased on information from an image sensor such as a camera and issuitably used when a vehicle control for assisting in a parkingoperation of a driver is performed depending on a sensing result will bedescribed with reference to the drawings.

First Embodiment

Hereinafter, a vehicle-mounted image processing device according to afirst embodiment of the present invention will be described withreference to FIGS. 1 to 14. FIG. 1 is a block diagram of avehicle-mounted image processing device 1000 according to the firstembodiment.

The vehicle-mounted image processing device 1000 is a device which isembedded in a camera device mounted on a vehicle, an integratedcontroller, or the like, and senses an object in an image captured bycameras 1001 to 1004 of the camera device. The vehicle-mounted imageprocessing device 1000 according to the present embodiment is configuredto sense a parking bay around a host vehicle 10 as a target object.

The vehicle-mounted image processing device 1000 is constituted by acomputer including a central processing unit (CPU), a memory, an I/O,and the like. A predetermined processing is programmed and is repeatedlyperformed in a predetermined cycle T.

As illustrated in FIG. 1, the vehicle-mounted image processing device1000 includes an image acquisition unit 1011, a white line featuresensing unit (feature amount extraction unit) 1021, a three-dimensionalobject sensing unit 1031, a three-dimensional object informationacquisition unit 1041, a vehicle behavior information acquisition unit1050, a three-dimensional object information storage unit 1051, a whiteline feature distinguishing unit (feature amount distinguishing unit)1061, and a parking bay recognition unit (target object recognitionunit) 1071.

The image acquisition unit 1011 acquires images 1011A, 1012A, 1013A, and1014A obtained by imaging an area around the host vehicle 10 by cameras(imaging unit) 1001, 1002, 1003, and 1004 attached at positions at whichthe cameras 1001, 1002, 1003, and 1004 can image the area around thehost vehicle 10 as illustrated in FIG. 2. Then, the acquired images1011A, 1012A, 1013A, and 1014A are geometrically transformed andsynthesized to generate an overhead image 1015 viewed from a virtualviewpoint above the host vehicle 10, and the overhead image 1015 isstored in a random access memory (RAM). A parameter for the geometrictransformation and synthesis is set in advance by calibration performedat the time of vehicle shipping. It should be noted that the overheadimage 1015 is a two-dimensional array and is represented byIMGSRC[x][y]. x and y each represent a coordinate of an image.

The white line feature sensing unit 1021 senses a white line featureLC[n] in the overhead image 1015 acquired by the image acquisition unit1011. The white line feature LC[n] includes information on coordinatesof a relative position of the white line feature to the host vehicle 10in the coordinate system (world coordinate) with the center of an axleof rear wheels of the host vehicle 10 as an origin, and is aone-dimensional array having a table as an element. n represents an IDwhen a plurality of white line features are sensed. Details of aprocessing therefor will be described later.

The three-dimensional object sensing unit 1031 senses a distance pointgroup IPT[b] which shows distance coordinates of a three-dimensionalobject around the host vehicle 10 by using the images 1011A, 1012A,1013A, and 1014A acquired by the camera 1001, 1002, 1003, and 1004,respectively. The distance point group IPT[b] is a one-dimensional arrayhaving a table including information such as distance coordinates of thethree-dimensional object as an element, and b represents an ID when aplurality of distance point groups IPT[b] are sensed. Details of aprocessing therefor will be described later.

The three-dimensional object information acquisition unit 1041 acquiresthree-dimensional object information, which is present around the hostvehicle 10 and is sensed by a sonar 1005 installed in the front, therear, and the side of the host vehicle 10, as a distance point groupSPT[c]. The distance point group SPT[c] is a one-dimensional arrayhaving a table including information such as distance coordinates of thethree-dimensional object or the like as an element, and c represents anID when a plurality of distance point groups SPT[c] are sensed.

The vehicle behavior information acquisition unit 1050 acquires avehicle behavior DRC calculated by using a pulse of a tire of the hostvehicle 10, or the like in an EC mounted in the host vehicle 10 or thevehicle-mounted image processing device 1000 through an in-vehiclenetwork CAN or the like. The vehicle behavior DRC includes informationof a speed (VX and VY) with respect to the world coordinate system and ayaw rate (YR).

The three-dimensional object information storage unit 1051 integratesand stores the distance point group IPT[b] sensed by thethree-dimensional object sensing unit 1031, the distance point groupSPT[c] acquired by the three-dimensional object information acquisitionunit 1041, and the vehicle behavior DRC acquired by the vehicle behaviorinformation acquisition unit 1050 as a three-dimensional object pointgroup OPT[d] including a past sensing result. Details of a processingtherefor will be described later.

The white line feature distinguishing unit 1061 distinguishes a whiteline LN[m] on a road surface among the white line features LC[n] byusing the white line feature LC[n] obtained by the white line featuresensing unit 1021, the three-dimensional object point group OPT[d]integrated and stored in the three-dimensional object informationstorage unit 1051, and camera geometric information CLB which isinformation on a coordinate position of the camera in the worldcoordinate system. Details of a processing therefor will be describedlater.

The parking bay recognition unit 1071 recognizes a parking bay by usingthe white line LN[m] distinguished as being present on the road surfaceby the white line feature distinguishing unit 1061. Details of aprocessing therefor will be described later.

Information of the parking bay recognized by the parking bay recognitionunit 1071 is output to another control unit or the like in the hostvehicle 10, other than the vehicle-mounted image processing device 1000,and is used when another control unit performs each control such as anautomatic driving control, an automatic parking control, a parkingassistance control, or the like.

[White Line Feature Sensing Unit 1021]

Next, contents of a processing performed by the white line featuresensing unit 1021 will be described with reference to FIG. 3. FIG. 3 isa flowchart illustrating a flow of a processing performed by the whiteline feature sensing unit 1021.

The white line feature sensing unit 1021 may perform a processing forthe entire overhead image 1015 or may define a processing region. In thepresent embodiment, an upper half portion of the overhead image 1015 isset as the processing region when the host vehicle is stopped or movesforward, and a lower half portion of the overhead image 1015 is set asthe processing region when the host vehicle moves backward, based on ashift position.

First, the overhead image 1015 is rotated by 90 degrees in step S301.

Then, in step S302, an edge filter in a lateral direction (a verticaldirection in the image before rotation) is applied for each line withina processing region while performing scanning from the left to the rightof the image. The following steps S303 and S304 are performed for eachline.

Next, an edge point at which an output value of the edge filter reachesa peak is sensed in step S303. A rising edge (a change point from darkto bright) Eu[nu] and a falling edge (a change point from bright todark) Ed[nd] are extracted as the peak, respectively.

The white line on the road surface has a brightness value higher than abrightness value of the road surface, and thus a rising edge is presentat the left side of the white line and a falling edge is present at theright side. In order to capture the feature, in step S304, only a pairof edges Ep[np], in which a falling edge is present within apredetermined range (a maximum thickness of the sensed white line whichis defined in advance) on the right side of the image from a risingedge, among rising edge points Eu[nu] and falling edge points Ed[nd]sensed in step S303, is left, and other single edges are regarded assingular points and thus are eliminated.

A series of the processing in steps S302 to S304 described above isperformed for each line within the processing region.

Then, among pairs of edges Ep[np] extracted in step S304, pairs of edgesEp[np] in which edges are aligned in a straight line are grouped,thereby generating a straight line candidate group Lg[ng] in step S305.Through this processing, pairs of edges in which edges are not alignedin a straight line are eliminated. Hough transform known in the art canbe used for grouping of the white line aligned in a straight line.

Then, in step S306, a filtering processing, in which a line with alength equal to or smaller than a predetermined length among lines ofthe straight line candidate group Lg[ng] is eliminated, is performed.

Then, in step S307, information of image coordinates of an upper end(start point) of a rising edge and a lower end (end point) of a fallingedge and coordinates of a relative portion to the host vehicle in theworld coordinate system which are calculated based on the overhead image1015 is stored as an element of the white line feature LC[n], the risingedge and the falling edge remaining in the group.

Here, although the case where the image is rotated in step S301 and aseries of the processing in steps S302 to S307 is performed for therotated image has been described, a processing of scanning the overheadimage 1015 without rotating the overhead image 1015 in a top-bottomdirection, and using a filter for sensing an edge in a lateral directioncan also be performed.

[Three-Dimensional Object Sensing Unit 1031]

Next, contents of a processing performed by the three-dimensional objectsensing unit 1031 will be described with reference to FIGS. 4 to 5B.FIG. 4 is a flowchart illustrating a flow of a processing performed bythe three-dimensional object sensing unit 1031. FIGS. 5A and 5B arediagrams for describing a processing performed by the three-dimensionalobject sensing unit 1031.

The three-dimensional object sensing unit 1031 performs a processing forany one or more of images 1011A, 1012A, 1013A, and 1014A. That is, thethree-dimensional object sensing unit 1031 may perform a processing foronly one image or may perform a processing for all the images. Accordingto the present embodiment, the three-dimensional object sensing unit1031 performs a processing for an image in a movement directiondepending on a shift position of the host vehicle.

First, in step S401, feature points FPT[f] are extracted from a currentimage IMG_C which is a processing target. A known method such as theHarris corner method is used for the extraction of the feature pointsFPT[f].

Then, in step S402, a past image IMG_P before a predetermined time,which is acquired by a camera imaging the image from which the featurepoints FPT[f] are extracted, is acquired.

Next, in step S403, a corresponding position of each feature pointFPT[f] of the current image IMG_C in the past image IMG_P is calculatedby an optical flow method, and movement vectors FPT_VX[f] and FPT_VY[f]of each feature point are acquired. As the optical flow method, a knownmethod such as the Lucas-Kanade method is used.

Then, in step S404, a three-dimensional position of each feature pointFPT[f] around the host vehicle 10 is calculated by using the featurepoint FPT[f] and the movement vectors FPT_VX[f] and FPT_VY[f]. As amethod for the calculation, known means is used.

Finally, in step S405, the three-dimensional position of each featurepoint is converted into the coordinate system (world coordinate system)with the center of the axle of the rear wheels of the vehicle as theorigin and is stored as the distance point group IPT[b].

As illustrated in FIGS. 5A and 5B, the calculation is performed based ona principle in which a distance is measured by using a fact that as apast position of the host vehicle and a current position of the hostvehicle are changed, parallax occurs. Distances of the respectivefeature points in the image are measured. Therefore, a result ofperforming the measurement with respect to, for example, a parkedvehicle is obtained as a group of a plurality of points with worldcoordinates as illustrated in FIG. 5B.

[Three-Dimensional Object Information Storage Unit 1051]

Next, contents of a processing performed by the three-dimensional objectinformation storage unit 1051 will be described with reference to FIGS.6 and 7A to 7D. FIG. 6 is a flowchart illustrating a flow of aprocessing performed by the three-dimensional object information storageunit 1051. FIGS. 7A to 7D are diagrams for describing a processingperformed by the three-dimensional object information storage unit 1051.

The three-dimensional object information storage unit 1051 stores thedistance point group IPT[b] calculated by the three-dimensional objectsensing unit 1031 and the distance point group SPT[c] of the sonaracquired by the three-dimensional object information acquisition unit1041, including past values thereof.

According to the present embodiment, the three-dimensional objectinformation storage unit 1051 manages all obstacle information with atwo-dimensional map EMP which has a certain position as an origin anddoes not have height information. The three-dimensional objectinformation storage unit 1051 attaches information of the distance pointgroup IPT[b] and the distance point group SPT[c] of the sonar which aresequentially calculated to a blank two-dimensional map by using thevehicle behavior DRC acquired by the vehicle behavior informationacquisition unit 1050, thereby creating a two-dimensional map EMP[x][y].Here, the EMP is a two-dimensional array and x and y are coordinates ofthe array partitioned with spatial resolution.

First, in step S601, a two-dimensional map EMP which is previouslyprocessed is acquired.

Then, in step S602, the vehicle behavior DRC is acquired from thevehicle behavior information acquisition unit 1050.

Then, in step S603, the distance point group IPT[b] calculated by thethree-dimensional object sensing unit 1031 is acquired.

Then, in step S604, the distance point group SPT[c] of the sonaracquired by the three-dimensional object information acquisition unit1041 is acquired.

The acquired distance point group IPT[b] or distance point group SPT[c]has information of relative coordinates to the host vehicle 10.Therefore, in step S605, the distance point group IPT[b] obtained by thecamera and the distance point group SPT[c] obtained by the sonar aremapped to a blank map, respectively, by using the vehicle behavior DRC,thereby creating the two-dimensional map EMP.

In addition, in step S606, point group information acquired in the pastis eliminated from all three-dimensional object point groups OPT[d]mapped to the two-dimensional map EMP, and a point group remaining inthe map is the three-dimensional object point group OPT[d].

Here, a certainty value is set for each grid of the two-dimensional mapand it is determined that an obstacle is present only in a grid with acertainty value which is equal to or greater than a predeterminedthreshold value, such that it is possible to cancel noise of the sensingresult.

For example, the three-dimensional object information storage unit 1051can increase a certainty value when the past information or a pluralityof sensing results are simultaneously obtained in the same grid in stepS605, and can decrease certainty values of all grids by a predeterminedvalue in step S606. As a result, a certainty value of a grid in whichsensing results are duplicated is increased and a certainty value of agrid in which a sensing result cannot be repeatedly obtained isdecreased, such that old information is eliminated.

FIGS. 7A to 7D illustrate an example of the processing. When distancepoint groups IPT[b] and SPT[c] obtained at a point in time t are asillustrated in FIG. 7A, distance point groups IPT[b] and SPT[c] obtainedat a point in time t-1 are as illustrated in FIG. 7B, and distance pointgroups IPT[b] and SPT[c] obtained at a point in time t-2 are asillustrated in FIG. 7C, an example of a three-dimensional object pointgroup OPT[d] which is a result of integrating the distance point groupsby using the vehicle behaviors DRC at the point in time t, the point intime t-1, and the point in time t-2, respectively, is as illustrated inFIG. 7D.

[White Line Feature Distinguishing Unit 1061]

Next, contents of a processing performed by the white line featuredistinguishing unit 1061 will be described with reference to FIGS. 8 and9A to 9F. FIG. 8 is a flowchart illustrating a flow of a processingperformed by the white line feature distinguishing unit 1061. FIGS. 9Ato 9F are diagrams for describing a processing performed by the whiteline feature distinguishing unit 1061.

The white line feature distinguishing unit 1061 distinguishes whetherthe white line feature LC[n] is the white line LN[m] on the road surfaceor a bumper, a side sill, or a coating of an adjacent parked vehiclewhich is an object on a three-dimensional object by using the white linefeature LC[n] obtained by the white line feature sensing unit 1021,three-dimensional object information OPT[d] obtained by thethree-dimensional object information storage unit 1051, and cameramounting position information CLB.

First, in step S801, the three-dimensional object information OPT[d] isacquired.

Then, in step S802, the camera mounting position information CLB isacquired.

Then, in step S803, the white line features LC[n] are acquired.

Then, in step S804, the following series of processing in steps S805 toS807 is performed with respect to n=1 to N for all white line featuresLC[n] in which n=1 to N.

First, in step S805, a triangle region D in the two-dimensionalcoordinate system in which a height is not considered is calculatedbased on three points including a start point coordinate and an endpoint coordinate of the white line feature LC[n], and a camera mountingposition coordinate.

Then, in step S806, whether or not the three-dimensional objectinformation OPT[d] is present in the triangle region D is determined.When it is determined that the three-dimensional object informationOPT[d] is not present in the triangle region D, the processing proceedsto step S807, and the white line feature LC[n] is registered as thewhite line LN[m]. Whereas, when it is determined that thethree-dimensional object information OPT[d] is present in the triangleregion D, the white line feature LC[n] is not regarded as the white lineLN[m] and the next white line feature LC[n] is subjected to thedetermination processing.

Examples of a series of the processing in steps S805 to S807 will bedescribed with reference to FIGS. 9A to 9F. Here, three white linefeatures LC[n] will be described. FIG. 9A illustrates a result ofsensing white line features LC[n] in the overhead image 1015. FIG. 9Billustrates three-dimensional object information OPT[d], and FIG. 9Cillustrates a result in which the three-dimensional object informationOPT[d] and the white line features LC[n] are overlapped with each other.FIGS. 9D, 9E, and 9F illustrate a state in which whether or notrespective triangle regions D overlap with respective white linefeatures LC[n] is confirmed.

The three-dimensional object information OPT[d] and the white linefeature LC[n] are overlapped with each other as illustrated in FIG. 9Cby using the white line feature LC[n] as illustrated in FIG. 9A, thecamera mounting position information CLB at which the white line featureLC[n] is sensed, and a sensing result of the three-dimensional objectinformation OPT[d] as illustrated in FIG. 9B. Based on FIG. 9C, threetriangle regions D illustrated in FIGS. 9D, 9E, and 9F, respectively,are generated, and whether or not the three-dimensional objectinformation OPT[d] is present inside the respective triangle regions Dis determined.

In this example, since the three-dimensional object information ispresent inside the triangle region D generated as illustrated in FIG.9F, the white line feature LC[n] illustrated in FIG. 9F is eliminated,and the remaining two white line features LC[n] illustrated in FIGS. 9Dand 9E, respectively, are registered as white lines LN[m].

[Parking Bay Recognition Unit 1071]

Next, contents of a processing performed by the parking bay recognitionunit 1071 will be described with reference to FIG. 10. FIG. 10 is aflowchart illustrating a flow of a processing performed by the parkingbay recognition unit 1071.

The parking bay recognition unit 1071 searches and recognizes a parkingbay within which the host vehicle 10 can be parked by combiningregistered white lines LN[m].

First, in step S1001, two lines LN[mL] and LN[mR] are selected from thewhite lines LN[m].

Then, in step S1002, whether or not an angle difference el in anextending direction of the two white lines LN[mL] and LN[mR] selected instep S1001 is equal to or less than a predetermined value (Thθmax) isdetermined. In other words, whether or not the two white lines areapproximately in parallel to each other is determined in step S1002.When it is determined that the angle difference el is equal to or lessthan the predetermined value, the processing proceeds to step S1003, andwhen it is determined that the angle difference el is greater than thepredetermined value, it is determined that the selected two white linesLN[mL] and LN[mR] do not correspond to a combination of white linesconstituting a parking bay and thus the processing proceeds to stepS1006.

When the determination is affirmative in step S1002, the processingproceeds to step S1003, and whether or not an interval W between the twowhite lines LN[mL] and LN[mR] is within a predetermined range (ThWmin orgreater and ThWmax or less) is determined. That is, in step S1003,whether or not the two white lines are arranged at an interval at whichthe two white lines can be considered as two white lines constituting aparking bay. When it is determined that the interval W is within thepredetermined range, the processing proceeds to step S1004, and when itis determined that the interval W is not within the predetermined range,the processing proceeds to step S1006.

When the determination is affirmative in step S1003, the processingproceeds to step S1004, and whether or not a misalignment degree ABbetween lower ends of the two white lines is within a predeterminedrange (ThBmin or greater and ThBmax or less) is determined. Here, themisalignment degree between the lower ends of the two white linesdetermined in step S1004 will be described with reference to FIGS. 11Aand 11B.

As illustrated in FIG. 11A, in a case where a parking bay 23 is providedin a space of a parking lot so that the host vehicle 10 is parked inparallel to or perpendicularly to a movement direction (a top-bottomdirection and a left-right direction in FIG. 11A) of the host vehicle10, and parking bay lines 23L are drawn to correspond to the parking bay23, lower ends of the parking bay lines 23L are not misaligned with eachother.

However, as illustrated in FIG. 11B, depending on a parking lot, aparking bay 23 is provided so that the host vehicle 10 is parkedobliquely with respect to a movement direction (a top-bottom directionand a left-right direction in FIG. 11B) of the host vehicle 10, ratherthan being parked in parallel to or perpendicularly to the movementdirection of the host vehicle 10, and parking bay lines 23L are drawn tocorrespond to the parking bay 23 in some cases. In this case, lower endpositions of the parking bay lines 23L constituting the parking bay 23are misaligned with each other as indicated by a distance B. The lowerend positions of the parking bay line 23L are misaligned with each otherin order to prevent the parking bay line 23L on the right side of FIG.11B from protruding to a region in which the host vehicle 10 travels.

For example, an intersection angle ρ between a line segment, which isperpendicular to an extending direction of the parking bay line 23L onthe left side of FIG. 11B, extending from the lower end of the parkingbay line 23L and a line segment connecting the lower ends of the parkingbay lines 23L to each other is fixed to be any one of 0 degrees, 30degrees, and 45 degrees in an expressway service area or a parking area.In addition, a range of a value which can be obtained as a width W ofthe parking bay 23 is already determined. Therefore, according to thepresent embodiment, a value of the distance B which can be obtained whenthe intersection angle θ is any one of 0 degrees, 30 degrees, and 45degrees is stored in a memory (not illustrated) or the like in advance.

In step S1004, distances B calculated based on the selected LN[mL] andLN[mR] are compared with each other and whether a difference between thedistances B is within a predetermined range (ThBmin or greater andThBmax or less) is determined, such that whether or not the two whitelines are white lines (parking bay lines 23L) constituting one parkingbay 23 is determined. When it is determined that the difference iswithin the predetermined range, the processing proceeds to step S1005,and when it is determined that the difference is not within thepredetermined range, the processing proceeds to step S1006.

When the determination is affirmative in step S1004, the processingproceeds to step S1005, and coordinates of four corners of a rectangularparking bay 23 constituted by the two white lines LN[mL] and LN[mR] areregistered as position information PS[k] on a position of one parkingbay 23.

When step S1005 is performed, the processing proceeds to step S1006, andwhether or not the processing for any two white lines described above isperformed for all white lines (parking bay line) based on theinformation output from the white line feature distinguishing unit 1061is confirmed. When the determination is affirmative in step S1006, aresult obtained by the processing described above is output and theprocessing performed by the parking bay recognition unit 1071 ends. Whenthe determination is negative in step S1006, the processing returns tostep S1001, and the processing described above is performed for allcombinations of the white lines LN[m].

Next, an effect of the present embodiment will be described.

As described above, when sensing a white line by using a camera image,whether a feature is present on a three-dimensional object or present ona road surface is not distinguished only from the camera image.Therefore, if all features are assumed as being present on the roadsurface and world coordinates thereof are measured, a feature, which ispresent on a three-dimensional object, is measured as being presentfurther away than it is. Accordingly, in the vehicle-mounted imageprocessing device 1000 according to the first embodiment of the presentinvention described above, white line features of a parking bay aresensed, a processing of overlapping acquired three-dimensional objectinformation with a triangle constituted by a start point and an endpoint obtained based on the white line feature and a camera mountingposition is performed, whether a feature having a feature amount of atarget object is present on a three-dimensional object or present on aroad surface is determined, and the target object is recognized by usingthe feature having the feature amount on the road surface.

By this processing, in a case of a white line feature which is presenton a three-dimensional object, three-dimensional object information ispositioned within the triangle. On the contrary, in a case of a featurewhich is present on a road surface, there is no three-dimensional objectpositioned within the triangle. By using this characteristic, it ispossible to distinguish only a white line feature, which is present on aroad surface, among white line features and to recognize a parking bay.As a result, in a system assisting in a parking operation, it ispossible to avoid misrecognition of apart of a three-dimensional objectas a parking bay and prevent a situation in which a target parkingposition is misrecognized as a part of a three-dimensional object andthus the vehicle moves toward the three-dimensional object and collideswith the three-dimensional object.

[Modified Example of Three-Dimensional Object Acquisition Unit Accordingto First Embodiment]

Hereinafter, a plurality of modified examples of acquisition ofthree-dimensional object information according to the first embodimentwill be described.

FIG. 12 is a block diagram of a vehicle-mounted image processing device1100 of a first modified example according to the present embodiment.The vehicle-mounted image processing device 1100 illustrated in FIG. 12is an example in which the vehicle-mounted image processing device 1000illustrated in FIG. 1 does not include the three-dimensional objectsensing unit 1031. In this case, the three-dimensional objectinformation storage unit 1051 performs a processing only with a resultwhich is obtained by the three-dimensional object informationacquisition unit 1041. In other words, step S603 in FIG. 6 is notperformed. Other configurations and processing are the same as theconfigurations and the processing of each unit according to the firstembodiment.

FIG. 13 is a block diagram of a vehicle-mounted image processing device1200 of a second modified example according to the present embodiment.The vehicle-mounted image processing device 1200 illustrated in FIG. 13is an example in which the vehicle-mounted image processing device 1000illustrated in FIG. 1 does not include the three-dimensional objectinformation acquisition unit 1041. In this case, the three-dimensionalobject information storage unit 1051 performs a processing only with aresult which is obtained by the three-dimensional object sensing unit1031. In other words, step S604 in FIG. 6 is not performed. Otherconfigurations and processing are the same as the configurations and theprocessing of each unit according to the first embodiment.

FIG. 14 is a block diagram of a vehicle-mounted image processing device1300 of a third modified example according to the present embodiment.The vehicle-mounted image processing device 1300 illustrated in FIG. 14is an example in which the vehicle-mounted image processing device 1000illustrated in FIG. 1 includes a LiDAR (light detection and ranging orlaser imaging detection and ranging) 1006, and the three-dimensionalobject information acquisition unit 1041 acquires information obtainedby the LiDAR 1006. The LiDAR is a device which measures scattered lightwith respect to laser irradiation in the form of pulses and analyzes adistance to an object which is positioned far away or a property of theobject. The LiDAR 1006 can detect a three-dimensional object, and thusthe three-dimensional object sensing unit 1031 is not necessary. Otherconfigurations and processing are the same as the configurations and theprocessing of the first modified example according to the firstembodiment illustrated in FIG. 12.

Second Embodiment

A vehicle-mounted image processing device according to a secondembodiment of the present invention will be described with reference toFIGS. 15 to 17E. FIG. 15 is a block diagram illustrating a configurationof a vehicle-mounted image processing device 2000 according to thesecond embodiment.

In the following description, only differences from the vehicle-mountedimage processing device 1000 according to the first embodiment will bedescribed in detail, and the same component is denoted by the samereference numeral and a detailed description thereof will be omitted.The same applies to the following embodiments.

As illustrated in FIG. 15, the vehicle-mounted image processing device2000 is different from the vehicle-mounted image processing device 1000according to the first embodiment in regard to the fact that thevehicle-mounted image processing device 2000 includes a white linefeature distinguishing unit (feature amount distinguishing unit) 2061instead of the white line feature distinguishing unit 1061.

The vehicle-mounted image processing device 2000 is a device which isembedded in a camera device mounted on a vehicle, an integratedcontroller, or the like, and senses an object in an image captured bycameras 1001 to 1004. The vehicle-mounted image processing device 2000according to the present embodiment is configured to sense a parking bayas a target object.

The vehicle-mounted image processing device 2000 is constituted by acomputer including a central processing unit (CPU), a memory, an I/O,and the like. A predetermined processing is programmed and is repeatedlyperformed in a predetermined cycle.

[White Line Feature Distinguishing Unit 2061]

Contents of a processing performed by the white line featuredistinguishing unit 2061 will be described with reference to FIGS. 16and 17A to 17E. FIG. 16 is a flowchart illustrating a flow of aprocessing performed by the white line feature distinguishing unit 2061.FIGS. 17A to 17E are diagrams for describing a processing performed bythe white line feature distinguishing unit 2061.

The white line feature distinguishing unit 2061 distinguishes whether ornot a white line feature LC[n] is a white line LN[m] on a road surfaceby using the white line feature LC[n] obtained by a white line featuresensing unit 1021 and three-dimensional object information OPT[d]obtained by a three-dimensional object information storage unit 1051,without using camera mounting position information CLB.

First, in step S1601, the three-dimensional object information OPT[d] isacquired.

Then, in step S1602, a rectangle with a predetermined size is fitted toa point group of the three-dimensional object information OPT[d], andthe rectangle of which a degree of fitting is equal to or greater than apredetermined value is stored as an approximating rectangle RCT[f].There is a known method for performing rectangle approximation for apoint group, and thus a detailed description thereof will be omitted. Asthe predetermined size, for example, a size of the host vehicle is used.

Then, in step S1603, the white line features LC[n] are acquired.

Then, in step S1604, a series of processing in steps S1605 and S1606 isperformed with respect to n=1 to N for all white line features LC[n] inwhich n=1 to N.

First, in step S1605, whether or not apart of the white line featureLC[n] is present inside any rectangle RCT[ f] is determined. When it isdetermined that the white line feature LC[n] is not present inside therectangle RCT[f] and does not overlap with the rectangle, the processingproceeds to step S1606, and the white line feature LC[n] is registeredas a white line LN[m]. Whereas, when it is determined that the whiteline feature LC[n] is at least partially present inside the rectangleRCT[f] and overlaps with the rectangle, the white line feature LC[n] isnot regarded as the white line LN[m] and the next white line featureLC[n] is subjected to the processing.

Examples of a series of the processing in steps S1605 and S1606 will bedescribed with reference to FIGS. 17A to 17E. Here, three white linefeatures LC[n] will be described. FIG. 17A illustrates a result ofsensing white line features LC[n] from an overhead image. FIG. 17Billustrates a result with three-dimensional object information OPT[d]and an approximating rectangle RCT[f]. FIGS. 17C, 17D, and 17E arediagrams illustrating a state in which whether or not respectiveapproximating rectangles RCT[f] overlap with respective white linefeatures LC[n] is confirmed.

An approximating rectangle RCT[f] illustrated in FIG. 17B is overlappedwith a white line feature LC[n] illustrated in FIG. 17A and whether ornot the white line feature LC[n] is at least partially present insidethe approximating rectangle RCT[f] is determined.

In this example, since a white line feature LC[n] in FIG. 17E is presentin the rectangle, the white line feature LC[n] in FIG. 17E iseliminated, and the remaining two white line features LC[n] illustratedin FIGS. 17C and 17D, respectively, are registered as white lines LN[m].

Other configurations and operations are approximately the same as theconfigurations and the operations of the vehicle-mounted imageprocessing device 1000 according to the first embodiment, and thus adetailed description thereof will be omitted.

As described above, it is highly likely that an obstacle in parking bayrecognition in a parking lot is a parked vehicle. Therefore, thevehicle-mounted image processing device 2000 according to the secondembodiment fits a rectangle having a size corresponding to a vehicle,for example, a rectangle having a size of the host vehicle, to thethree-dimensional object information. By this processing, it is possibleto predict a position of an obstacle outside a sensing range anddetermine whether a white line feature is a feature on athree-dimensional object or a feature on a road surface by using theapproximating rectangle. Therefore, almost the same effect as the effectof the vehicle-mounted image processing device 1000 according to firstembodiment described above is obtained.

Third Embodiment

A vehicle-mounted image processing device according to a thirdembodiment of the present invention will be described with reference toFIGS. 18 to 23. FIG. 18 is a block diagram illustrating a configurationof a vehicle-mounted image processing device 3000 according to the thirdembodiment.

As illustrated in FIG. 18, the vehicle-mounted image processing device3000 according to the present embodiment includes an endpoint featuresensing unit (feature amount extraction unit) 3021, an end point featuredistinguishing unit (feature amount distinguishing unit) 3061, and aparking bay recognition unit (target object recognition unit) 3071,instead of the configuration of the vehicle-mounted image processingdevice 1000 according to the first embodiment for the white line featuresensing.

The vehicle-mounted image processing device 3000 is a device which isembedded in a camera device mounted on a vehicle, an integratedcontroller, or the like, and senses an object in an image captured bycameras 1001 to 1004, and the vehicle-mounted image processing device3000 according to the present embodiment is configured to sense aparking bay as a target object.

The vehicle-mounted image processing device 3000 is constituted by acomputer including a central processing unit (CPU), a memory, an I/O,and the like. A predetermined processing is programmed and is repeatedlyperformed in a predetermined cycle.

[End Point Feature Sensing Unit 3021]

Contents of a processing performed by the end point feature sensing unit3021 will be described with reference to FIGS. 19 to 20B. FIG. 19 is aflowchart illustrating a flow of a processing performed by the end pointfeature sensing unit 3021. FIGS. 20A and 20B are diagrams for describinga processing performed by the end point feature sensing unit 3021.

The end point feature sensing unit 3021 performs a processing of sensingend points in a current image IMG_C acquired by an image acquisitionunit 1011 and sensing end point features PC[n] by using a combination ofthe end points.

First, in step S1901, an overhead image 1015 acquired by the imageacquisition unit 1011 is acquired.

Then, in step S1902, feature points LPT[g] are sensed. In the presentembodiment, Harris feature points are sensed. (1) in FIG. 20A and (1) inFIG. 20B illustrate an example in which feature points LPT[g] aresensed.

Then, in step S1903, a surrounding pattern of a feature point LPT[g] isacquired. The end point feature sensing unit 3021 acquires a localpattern LPTN[g] from an image having a predetermined pixel of width anda predetermined pixel of height with the feature point as the center.The acquired pattern may be the image as it is, binarized information,or one which stores a rising angle and a falling angle of an edge.Examples of the acquired surrounding pattern are illustrated in (2) inFIG. 20A.

Then, in steps S1904 and S1905, a calculation is performed for thesensed feature points LPT[g], in which g1=1, . . . , G, and g2=g1+1, . .. , G, and the following series of processing of steps S1906 to S1908 isperformed for all combinations of LPT[g1] and LPT[g2].

First, in step S1906, whether or not the local patterns LPTN[g1] andLPTN[g2] each have a certain pattern shape is determined. The certainpattern is a pattern in which an angle of a parking bay desired to besensed is defined. For example, the certain pattern is a pattern shapeas illustrated in (3) in FIG. 20A or (2) in FIG. 20B

(4) in FIG. 20A and (3) in FIG. 20B illustrate a result which isobtained by the determination based on (1) in FIG. 20A and (1) in FIG.20B, respectively, and in which only LPT[g] with the certain patternremains. Here, in a case where the local patterns LPTN[g1] and LPTN[g2]satisfy the condition, the processing proceeds to step S1907, and in acase where the local patterns LPTN[g1] and LPTN[g2] do not satisfy thecondition, a subsequent processing is not performed and a loopprocessing is continuously performed while incrementing g1 and g2.

Then, in step S1907, the local patterns LPTN[g1] and LPTN[g2] arecompared in regard to symmetry. The comparison is performed by, forexample, generating a pattern LPTN′ [g2] obtained by reversing the localpattern LPTN[g2] with a vertical line orthogonal to a line segmentconnecting two selected points as a symmetry axis and calculating asameness between LPTN[g1] and LPTN′[g2]. (5) in FIG. 20A and (4) in FIG.20B illustrate examples of the calculation. Then, whether or not asameness between the local patterns LPTN[g1] and LPTN[g2] is equal to orgreater than a predetermined threshold value is determined, and in acase where the sameness is equal to or greater than the predeterminedthreshold value, the processing proceeds to step S1908, and in a casewhere the sameness is not equal to or greater than the predeterminedthreshold value, a subsequent processing is not performed and a loopprocessing is continuously performed while incrementing g1 and g2.

Then, in step S1908, a line segment including LPT[g1] and LPT[g2] isregistered as an end point feature PC[n]. (6) in FIG. 20A and (5) inFIG. 20B are examples of the line segment PC[n] registered as a resultof the processing described above.

[End Point Feature Distinguishing Unit 3061]

Next, contents of a processing performed by the end point featuredistinguishing unit 3061 will be described with reference to FIGS. 21Ato 21D. FIGS. 21A to 21D are diagrams for describing a processingperformed by the endpoint feature distinguishing unit 3061.

The end point feature distinguishing unit 3061 distinguishes whether ornot an endpoint feature PC[n] is an endpoint line segment PN[m] on aroad surface by using the endpoint feature PC[n] obtained by the endpoint feature sensing unit 3021, three-dimensional object informationOPT[d] obtained by a three-dimensional object information storage unit1051, and camera mounting position information CLB.

A flow of the processing performed by the end point featuredistinguishing unit 3061 is the same as the processing performed by thewhite line feature distinguishing unit 1061 described in FIG. 8, exceptthat an input is changed from the white line feature LC[n] to the endpoint feature PC[n] and an output is changed from the white line LN[m]to the end point line segment PN[m], and thus a description thereof willbe omitted.

An example of the processing performed by the end point featuredistinguishing unit 3061 will be described with reference to FIGS. 21Ato 21D. First, FIG. 21A is an example in which end point features PC[n]are sensed, and two end point features PC[n] are sensed. FIG. 21B is anexample in which the end point features PC[n] and the three-dimensionalobject information OPT[d] are overlapped. FIGS. 21C and 21D illustrate astate in which whether or not respective end point features PC[n]overlap with respective three-dimensional object information OPT[d] isconfirmed.

In the processing performed by the end point feature distinguishing unit3061, a triangle is generated by the two end point features PC[n] andthe camera mounting position information CLB as illustrated in FIGS. 21Cand 21D, respectively, and whether or not the three-dimensional objectinformation OPT[d] is present inside the triangle is confirmed. Here,the end point features PC[n] illustrated in FIG. 21C are not within thetriangle and the end point features PC[n] illustrated in FIG. 21D arewithin the triangle. Therefore, the endpoint features PC[n] in FIG. 21Care registered as an endpoint line segment PN[m] on the road surface.

[Parking Bay Recognition Unit 3071]

Next, contents of a processing performed by the parking bay recognitionunit 3071 will be described with reference to FIGS. 22 and 23. FIG. 22is a flowchart illustrating a flow of a processing performed by theparking bay recognition unit 3071.

The parking bay recognition unit 3071 searches and recognizes a parkingbay within which the host vehicle 10 can be parked by combiningregistered end point line segments PN[m].

First, in step S2201, one end point line segment PN[m] is selected.

Then, in step S2202, a standard azimuth BB is acquired. Here, as thestandard azimuth 613, various azimuths can be used. For example, anazimuth of the host vehicle may be used, and a bearing of a trajectoryon which the host vehicle travels for a predetermined time may also beused. In addition, an average bearing of a short side of a past sensingresult of a parking bay may be used, and an average value of a shortside of the rectangle fitted to the obstacle OPT[d] in the processingperformed by the white line feature distinguishing unit 2061 accordingto the second embodiment may also be used.

Then, in step S2203, whether or not an angle difference el between anangle of the end point line segment PN[m] selected in step S2201 and thestandard azimuth θB is equal to or less than a predetermined value(Thθmax) is determined. In other words, whether or not the end pointline segment is in parallel to the standard azimuth is determined instep S2203. When it is determined that the angle difference θ is equalto or less than the predetermined value, the processing proceeds to stepS2204, and when it is determined that the angle difference θ is greaterthan the predetermined value, the processing proceeds to step S2206.

When the determination is affirmative in step S2203, the processingproceeds to step S2204, and whether or not a length L of the endpointline segment PN[m] is within a predetermined range (ThLmin or greaterand ThLmax or less) is determined. In other words, whether or not aninterval of the end point line segment constituting a short side of aparking bay corresponds to a width of the parking bay is determined instep S2204. When it is determined that the length L is within thepredetermined range, the processing proceeds to step S2205, and when itis determined that the length L is not within the predetermined range,the processing proceeds to step S2206.

When the determination is affirmative in step S2204, the processingproceeds to step S2205, and coordinates of four corners of a rectangularparking bay with the endpoint line segment PN[m] as a short side areregistered as position information PS[k]. Here, coordinates obtainedbased on the endpoint line segment correspond to two points in the frontof the parking bay, and end point positions within the parking bay arenot sensed, and thus interpolation is performed by using a length of thehost vehicle, or the like.

When step S2205 is performed, the processing proceeds to step S2206, andwhether or not the processing described above is performed for allendpoint features (an entrance line of a parking bay) based on theinformation output from the end point feature distinguishing unit 3061is confirmed. When the determination is affirmative in step S2206, aresult obtained by the processing described above is output and theprocessing performed by the parking bay recognition unit 3071 ends. Whenthe determination is negative in step S2206, the processing returns tostep S2201.

Contents of a processing of recognizing a parking bay, which isdifferent from the processing performed by the parking bay recognitionunit 3071 described above will be described with reference to FIG. 23.FIG. 23 is a flowchart illustrating a flow of another processingperformed by the parking bay recognition unit 3071.

First, in step S2301, two end point line segments PN[m] are selected.Hereinafter, the end point line segments PN[m] are end point linesegments PN[m1] and PN[m2].

Then, in step S2302, whether or not a length difference ΔL between thetwo endpoint line segments PN[m1] and PN[m2] selected in step S2301 iswithin a predetermined range (ThΔLmin or greater and ThΔLmax or less) isdetermined. In other words, whether or not there is another line segmentwhich has a length similar to a length of an end point line segmentconstituting a short side of a parking bay is confirmed in step S2302.When it is determined that the length difference ΔL is within thepredetermined range, the processing proceeds to step S2303, and when itis determined that the length difference ΔL is not within thepredetermined range, the processing proceeds to step S2305.

When the determination is affirmative in step S2302, the processingproceeds to step S2303, and whether or not an angle difference Δθbetween the endpoint line segments PN[m1] and PN[m2] selected in stepS2301 is equal to or less than a predetermined value (Thθmax) isdetermined. In other words, whether or not the two end point linesegments are in parallel to each other is determined in step S2303. Whenit is determined that the angle difference Δθ is equal to or less thanthe predetermined value, the processing proceeds to step S2304, and whenit is determined that the angle difference Δθ is greater than thepredetermined value, the processing proceeds to step S2305.

When the determination is affirmative in step S2303, the processingproceeds to step S2304, and coordinates of four corners of a rectangleparking bay with the endpoint line segment PN[m1], which is a standard,as a short side are registered as position information PS[k]. Here,coordinates obtained based on the end point line segment correspond totwo points in the front of the parking bay, and end point positionswithin the parking bay are not sensed, and thus interpolation isperformed by using a length of the host vehicle, or the like.

When step S2304 is performed, the processing proceeds to step S2305, andwhether or not the processing described above is performed for allendpoint features (an entrance line of a parking bay) based on theinformation output from the end point feature distinguishing unit 3061is confirmed. When the determination is affirmative in step S2305, aresult obtained by the processing described above is output and theprocessing performed by the parking bay recognition unit 3071 ends. Whenthe determination is negative in step S2305, the processing returns tostep S2301.

In the processing in FIG. 23, in a case where two or more parking baysare sensed, it is possible to further reduce misrecognition incomparison to a case where the determination is performed only with asingle end point line segment as in the processing in FIG. 22.

Other configurations and operations are approximately the same as theconfigurations and the operations of the vehicle-mounted imageprocessing device 1000 according to the above-described firstembodiment, thus a detailed description thereof will be omitted.

As described above, when a parking bay, in which paintings are presentonly at four corners of the parking bay illustrated in (1) in FIG. 20Aand (1) in FIG. 20B, is sensed, a situation, in which a pattern ofstopped vehicles on the left and the right of the parking bay, such as alicense plate, is misrecognized as a part of the parking bay, occurs asillustrated in FIG. 21D. In this regard, the misrecognition can beeliminated in the vehicle-mounted image processing device 3000 accordingto the third embodiment, and almost the same effect as the effect of thevehicle-mounted image processing device 1000 according to the firstembodiment described above can be obtained.

Fourth Embodiment

A vehicle-mounted image processing device according to a fourthembodiment of the present invention will be described with reference toFIGS. 24 and 25. FIG. 24 is a block diagram illustrating a configurationof a vehicle-mounted image processing device 4000 according to thefourth embodiment.

As illustrated in FIG. 24, the vehicle-mounted image processing device4000 according to the present embodiment includes a road surfacepainting recognition unit (target object recognition unit) 4071, insteadof the parking bay recognition unit 1071 of the vehicle-mounted imageprocessing device 1000 according to the first embodiment.

The vehicle-mounted image processing device 4000 is a device which isembedded in a camera device mounted on a vehicle, an integratedcontroller, or the like, and senses an object in an image captured bycameras 1001 to 1004, and the vehicle-mounted image processing device4000 according to the present embodiment is configured to sense a roadsurface painting as a target object.

The vehicle-mounted image processing device 4000 is constituted by acomputer including a central processing unit (CPU), a memory, an I/O,and the like. A predetermined processing is programmed and is repeatedlyperformed in a predetermined cycle.

[Road Surface Painting Recognition Unit 4071]

Contents of a processing performed by the road surface paintingrecognition unit 4071 will be described with reference to FIG. 25. FIG.25 is a flowchart showing a processing performed by the road surfacepainting recognition unit 4071.

The road surface painting recognition unit 4071 recognizes a pattern ofa painting on a road surface by using LN[m] obtained by the white linefeature distinguishing unit 1061. In the present embodiment, aprocessing of recognizing a crosswalk will be described.

First, in step S2501, one white line LN[m1] as a standard is selected.

Then, a reference white line LN[m2] is selected in step S2502, and aseries of processing in steps S2503 to S2505 is performed for all whitelines other than m1.

First, in step S2503, whether or not an angle difference el between thewhite line LN[m1] as a standard selected in step S2501 and the referencewhite line LN[m2] is equal to or less than a predetermined value(Thθmax) is determined. In other words, whether or not the two whitelines are in parallel to each other is determined in step S2503. When itis determined that the angle difference el is equal to or less than thepredetermined value, the processing proceeds to step S2504, and when itis determined that the angle difference el is greater than thepredetermined value, a loop processing is continuously performed.

When the determination is affirmative in step S2503, the processingproceeds to step S2504, and whether or not an interval W between the twowhite lines LN[m1] and LN[m2] is within a predetermined range (ThWmin orgreater and ThWmax or less) is determined. In other words, whether ornot the two white lines are arranged at an interval corresponding to thecrosswalk is determined in step S2504. When it is determined that theinterval W is within the predetermined range, the processing proceeds tostep S2505, and when it is determined that the interval W is not withinthe predetermined range, a loop processing is continuously performed.

Here, in the determination of the interval W, whether or not each ofW/2, W/3, W/4, . . . is within the predetermined range is alsodetermined. When any white line of the crosswalk is selected, aninterval between the selected white line and an adjacent white line iswithin the predetermined range. However, when the next white linedisposed after the adjacent white line adjacent to the selected whiteline is selected, the interval doubles and when a subsequent white linedisposed after the next white line is selected, the interval triples.Therefore, it is determined that the interval W is within thepredetermined range also in a case where any of the conditions describedabove is satisfied.

When the determination is affirmative in step S2504, the processingproceeds to S2505, and a crosswalk score PCR[m1] is incremented.

After performing the processing for all reference white lines LN[m2],the processing proceeds to step S2506, and whether the crosswalk scorePCR[m1] is equal to or greater than a predetermined threshold value isdetermined. When the crosswalk score PCR[m1] is equal to or greater thanthe predetermined threshold value, the processing proceeds to stepS2507. A coordinate values or a flag of the sensing result, or the likeis registered as a sensing result, and when it is determined that thecrosswalk score PCR[m1] is less than the threshold value, the nextstandard white line is selected and a loop processing is continuouslyperformed to determine the crosswalk score PCR[m] with all white lines.

The determined information of the crosswalk is output to another controlunit in the host vehicle 10, other than the vehicle-mounted imageprocessing device 4000, and is used when another control unit performseach control such as an automatic driving control.

Other configurations and operations are approximately the same as theconfigurations and the operations of the vehicle-mounted imageprocessing device 1000 according to the above-described firstembodiment, thus a detailed description thereof will be omitted.

As described above, also when sensing a road surface painting, it ispossible to eliminate a feature which is similar to the road surfacepainting and is present at an inner side of an obstacle by thevehicle-mounted image processing device 4000 according to the fourthembodiment, thereby making it possible to correctly recognize the roadsurface painting.

According to the fourth embodiment, the white line feature LC[n] sensedby the white line feature sensing unit 1021 and the white line LN[m]distinguished by the white line feature distinguishing unit 1061, whichare used in the first embodiment, are used as the feature of the roadsurface painting. However, as described in the third embodiment, an endpoint feature PC[n] sensed by the end point feature sensing unit 3021and an end point line segment PN[m] distinguished by the end pointfeature distinguishing unit 3061 can also be used.

In addition, although the crosswalk has been described by way of examplein the present embodiment, another road surface painting (an arrow, aspeed mark, a stop sign, or the like) can also be recognized by changinga condition of the road surface painting recognition unit 4071.

Fifth Embodiment

A vehicle-mounted image processing device according to a fifthembodiment of the present invention will be described with reference toFIGS. 26 and 27. FIG. 26 is a block diagram illustrating a configurationof a vehicle-mounted image processing device 5000 according to the fifthembodiment.

As illustrated in FIG. 26, the vehicle-mounted image processing device5000 according to the present embodiment is different from thevehicle-mounted image processing device 1000 according to the firstembodiment in regard to the fact that the vehicle-mounted imageprocessing device 5000 includes a curbstone feature sensing unit(feature amount extraction unit) 5021, a curbstone featuredistinguishing unit (feature amount distinguishing unit) 5061, and acurbstone recognition unit (target object recognition unit) 5071, andfurther includes a parking space acquisition unit 5081.

The vehicle-mounted image processing device 5000 is a device which isembedded in a camera device mounted on a vehicle, an integratedcontroller, or the like, and senses an object in an image captured bycameras 1001 to 1004, and the vehicle-mounted image processing device5000 according to the present embodiment is configured to sense acurbstone as a target object. The curbstone as a target object in thepresent embodiment is present near a region in which a plurality ofparking bays are arranged, lies in parallel to a bay line, and isprovided in parallel to a white line for parallel parking which isprovided on a road.

The vehicle-mounted image processing device 5000 is constituted by acomputer including a central processing unit (CPU), a memory, an I/O,and the like. A predetermined processing is programmed and is repeatedlyperformed in a predetermined cycle.

The curbstone feature sensing unit 5021 senses a line segment as afeature of the curbstone in an image IMG_C acquired by the imageacquisition unit 1011. The curbstone feature sensing unit 5021 detectsedges from the image by using a difference between a brightness of thecurbstone and a brightness of a road surface, and senses a line segmentby using an interval between a rising edge and a falling edge. Thisprocessing may be the same as the processing performed by the white linefeature sensing unit 1021 used in the first embodiment, and thus adetailed description thereof will be omitted.

Similarly, a processing performed by the curbstone featuredistinguishing unit 5061 may be the same as the processing performed bythe white line feature distinguishing unit 1061 in the first embodiment,and thus a detailed description thereof will be omitted.

[Parking Space Acquisition Unit 5081]

The parking space acquisition unit 5081 recognizes a parking spacearound the host vehicle 10 and acquires rectangle information PS[p] ofthe parking space. The rectangle information PS[p] may be acquired as,for example, a result of parking bay recognition executed in the firstembodiment described above, may be acquired by regarding a spacegenerated between three-dimensional object point group OPT[d] in thethree-dimensional object information storage unit 1051 as a parking bay,and can be acquired by known means. This processing can be executed byknown means, a description thereof will be omitted.

[Curbstone Recognition Unit 5071]

Contents of a processing performed by the curbstone recognition unit5071 will be described with reference to FIG. 27. FIG. 27 is a flowchartshowing a processing performed by the curbstone recognition unit 5071.

The curbstone recognition unit 5071 recognizes a curbstone by using aline segment LN[m] obtained by the curbstone feature distinguishing unit5061 and a parking space PS[p].

First, in step S2701, a line segment LN[m] is selected.

Then, a parking space PS[p] is selected in step S2702, and a series ofprocessing in steps S2703 to S2705 is performed for all parking spacesPS[p].

First, in step S2703, whether or not an angle difference el between theline segment LN[m] selected in step S2701 and the parking space PS [p]is equal to or less than a predetermined value (Thθmax) is determined.In other words, whether or not the line segment is in parallel to theparking space is determined in step S2703. When it is determined thatthe angle difference θ is equal to or less than the predetermined value,the processing proceeds to step S2704, and when it is determined thatthe angle difference θ is greater than the predetermined value, a loopprocessing is continuously performed.

When the determination is affirmative in step S2703, the processingproceeds to step S2704, and whether or not an interval W between theline segment LN[m] and a long side of the parking space PS[p], which ispositioned far away from the host vehicle, is equal to or less than apredetermined value (ThWmax) is determined. In other words, whether ornot the line segment is present inside the parking space is determinedin step S2704. When it is determined that the interval W is equal to orless than the predetermined value, the processing proceeds to stepS2705, and when it is determined that the interval W is greater than thepredetermined value, a loop processing is continuously performed.

When the determination is affirmative in step S2704, the processingproceeds to step S2705, and whether or not a point group of an objectwith a height equal to or less than a predetermined value is present onthe line segment LN[m] is checked with reference to thethree-dimensional object information OPT[d]. When it is determined thatthe object is present, the processing proceeds to step S2706, and whenit is determined that the object is not present, a loop processing iscontinuously performed.

When the determination is affirmative in step S2705, the processingproceeds to S2706, and the line segment is registered as a curbstone.Registration information is information of a start point and an endpointof the curbstone, that is, the line segment.

Although the case where a target object to be sensed is a curbstone hasbeen described, the target object to be sensed can also be a wheelstopper in a parking bay. In this case, whether or not a bearing of aparking bay and an angle of a wheel stopper are orthogonal to each otheris determined in step S2703, and an interval between the wheel stopperand a short side of the parking space PS[p], which is positioned faraway from the host vehicle, is determined in step S2704.

Other configurations and operations are approximately the same as theconfigurations and the operations of the vehicle-mounted imageprocessing device 1000 according to the first embodiment, thus adetailed description thereof will be omitted.

As described above, even in a case where an object with a small heightsuch as a curbstone or a wheel stopper is sensed based on edge featuresin an image, it is possible to perform sensing with high accuracywithout erroneous sensing by the vehicle-mounted image processing device5000 according to the fifth embodiment, and almost the same effect asthe effect of the vehicle-mounted image processing device 1000 accordingto the first embodiment described above can be obtained.

<Others>

It should be noted that the present invention is not limited to theembodiments described above, but includes various modified examples. Theembodiments described above have been described in detail in order tofacilitate understanding of the present invention, and are notnecessarily limited to including all the configurations described above.In addition, a part of the configuration of one embodiment can bereplaced with the configuration of another embodiment or theconfiguration of another embodiment can be added to the configuration ofone embodiment. Further, it is possible to add, delete, and replaceother configurations with respect to a part of the configuration of eachembodiment.

REFERENCE SIGNS LIST

-   10 host vehicle-   23 parking bay-   23L parking bay line-   1000,1100,1200,1300,2000,3000,4000,5000 vehicle-mounted image    processing device-   1001,1002,1003,1004 camera (imaging unit)-   1005 sonar-   1006 LiDAR-   1011 image acquisition unit-   1011A,1012A,1013A,1014A image-   1015 overhead image-   1021 white line feature sensing unit (feature amount extraction    unit)-   1031 three-dimensional object sensing unit-   1041 three-dimensional object information acquisition unit-   1050 vehicle behavior information acquisition unit-   1051 three-dimensional object information storage unit-   1061 white line feature distinguishing unit (feature amount    distinguishing unit)-   1071 parking bay recognition unit (target object recognition unit)-   2061 white line feature distinguishing unit (feature amount    distinguishing unit)-   3021 end point feature sensing unit (feature amount extraction unit)-   3061 end point feature distinguishing unit (feature amount    distinguishing unit)-   3071 parking bay recognition unit (target object recognition unit)-   4071 road surface painting recognition unit (target object    recognition unit)-   5021 curbstone feature sensing unit (feature amount extraction unit)-   5061 curbstone feature distinguishing unit (feature amount    distinguishing unit)-   5071 curbstone recognition unit (target object recognition unit)-   5081 parking space acquisition unit

The invention claimed is:
 1. A vehicle-mounted image processing devicerecognizing a target object around a host vehicle, the vehicle-mountedimage processing device comprising: a central processing unit; and amemory coupled to the central processing unit and storing a programwhich, when executed by the central processing unit, causes the centralprocessing unit to acquire an image around the host vehicle captured bya camera; extract, from the image around the host vehicle acquired bythe camera, a feature amount of the target object and coordinateinformation of the feature amount with respect to the host vehicle whenit is assumed that a feature having the feature amount is on a roadsurface; acquire and store coordinate information of a three-dimensionalobject around the host vehicle with respect to the host vehicle;distinguish whether the feature amount is a road surface feature amountof a feature on the road surface or a three-dimensional object featureamount of a feature on the three-dimensional object by using apositional relationship between the coordinate information of thefeature amount with respect to the host vehicle and the coordinateinformation of the three-dimensional object stored with respect to thehost vehicle; recognize the target object by using the road surfacefeature amount; sense a line segment as a line segment feature amount,and calculate coordinate information of a start point and an end pointof the line segment with respect to the host vehicle, as the coordinateinformation of the feature amount with respect to the host vehicle;distinguish whether the line segment feature amount is the road surfacefeature amount or the three-dimensional object feature amount by using apositional relationship between the coordinate information of the startpoint and the end point of the line segment with respect to the hostvehicle and the coordinate information of the three-dimensional objectwith respect to the host vehicle; and calculate a triangle including amounting position of the camera at which the feature amount isextracted, the start point of the line segment, and the end point of theline segment, determines whether or not the coordinate information ofthe three-dimensional object with respect to the host vehicle is presentinside the triangle, determines that the feature amount is the roadsurface feature amount when it is determined that the coordinateinformation of the three-dimensional object with respect to the hostvehicle is not present inside the triangle, and determines that thefeature amount is the three-dimensional object feature amount when it isdetermined that the coordinate information of the three-dimensionalobject with respect to the host vehicle is present inside the triangle.2. The vehicle-mounted image processing device according to claim 1,wherein the central processing unit senses edges from the image andsenses, as the line segment, a line segment connecting a start point andan end point of the edges aligned in a straight line.
 3. Thevehicle-mounted image processing device according to claim 1, theprogram further causing the central processing unit to: acquire thecoordinate information of the three-dimensional object with respect tothe host vehicle; and acquire behavior information of the host vehicle,and convert three-dimensional object information acquired in a past intorelative coordinates from a current host vehicle position by using thecoordinate information of the three-dimensional object with respect tothe host vehicle and the behavior information of the host vehicle, andstore accumulated information as coordinate information of thethree-dimensional object around the host vehicle with respect to thehost vehicle.
 4. The vehicle-mounted image processing device accordingto claim 1, wherein the target object is a parking bay, and the centralprocessing unit selects two line segments which have features havingfeature amounts determined as feature amounts of features on the roadsurface, and recognizes the parking bay by using conditions including aninterval and an angle difference between the two line segments.
 5. Thevehicle-mounted image processing device according to claim 1, whereinthe central processing unit performs a rectangle approximation for thecoordinate information of the three-dimensional object with respect tothe host vehicle, determines whether or not the line segment is at leastpartially present inside the coordinate information of thethree-dimensional object with respect to the host vehicle subjected tothe rectangle approximation, determines that the feature amount is theroad surface feature amount when it is determined that the line segmentis not at least partially present inside the coordinate information ofthe three-dimensional object with respect to the host vehicle subjectedto the rectangle approximation, and determines that the feature amountis the three-dimensional object feature amount when it is determinedthat the line segment is at least partially present inside thecoordinate information of the three-dimensional object with respect tothe host vehicle subjected to the rectangle approximation.
 6. Thevehicle-mounted image processing device according to claim 1, whereinthe central processing unit senses corner points from the image andsenses, as a start point and an end point, a line segment with acombination of corner points, of which symmetry between image patternsaround the corner points is equal to or greater than a predeterminedstandard value, among two combinations of the corner points.
 7. Thevehicle-mounted image processing device according to claim 6, whereinthe target object is a parking bay, and the central processing unitselects one line segment which is the road surface feature amount, andrecognizes the parking bay by using a condition including a length ofthe selected line segment.
 8. The vehicle-mounted image processingdevice according to claim 1, wherein the target object is a road surfacepainting, and the central processing unit selects a plurality of linesegments which have features having feature amounts determined as roadsurface feature amounts of features on the road surface, and recognizesthe road surface painting by using a positional relationship among theplurality of selected line segments.
 9. The vehicle-mounted imageprocessing device according to claim 2, wherein the target object is acurbstone or a wheel stopper, the central processing unit acquires aparking space around the host vehicle, and the central processing unitrecognizes the target object by using a bearing relationship between thefeature amount determined as a feature amount of a feature on the roadsurface and the parking space, and a positional relationship between theroad surface feature amount and the parking space.